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Abstract 

A subcategory of computer-assisted instruction ( CAI), games have additional attributes such as 
motivation, reward, interactivity, score, and challenge. This study used a quasi-experimental 
design to determine if previous findings generalize to non simidation-based game designs. 
Researchers observed significant improvement in the overall population for math skills in the 
non-game CAI control condition, but not in the game-based experimental condition. The 
study found no meaningfid, significant differences in language arts skills in any of the condi- 
tions. This finding has implications for the design of future learning games, suggesting that 
a simulation-based approach shoidd be integrated into the gaming technology. (Keywords: 
games, education, simulation, CAI, motivation.) 

INTRODUCTION 

Computer-assisted instruction is a widely studied and supported method 
of teaching. Numerous meta-analyses and review articles have been published 
showing small but positive effect sizes supporting CAI over other teaching 
methods (Bayraktar, 2001; Chambers, 2002; Christmann & Badgett, 2003; 
Cohen & Dacanay, 1992; Fletcher-Flinn & Gravatt, 1995; Kulik, 1994; Lowe, 
2001). CAI is defined as any program that augments, teaches, or simulates the 
learning environment used in the traditional classroom (Quyang, 1993), in- 
cluding Web-based instruction, self-running simulations, drill-and-practice pro- 
grams, and multimedia classrooms (Murphy et al., 2002). Learning games are 
just one form of CAI that have the following additional attributes: motivation, 
reward (feedback), interactivity, score, and challenge. Support also exists for this 
specific type of CAI and its effectiveness in the classroom (Vogel et al., 2006). 
However, it remains unclear as to whether or not learning through games will 
improve upon traditional CAI results. 

Motivation to Learn 

The primary thrust of the use of games within the domain of CAI is to in- 
crease students’ motivation to learn, by presenting the learning material in a 
form that encourages engagement and thereby increases practice. Research 
suggests that interacting with computers leads to increased motivation (Ju & 
Wagner, 1997; Kafai, 2001; Rieber, 1996). Specifically, environments such as 
games that have increased learner-control and learner-created feedback over 
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traditional computer-centered CAI result in higher levels of motivation (Nas- 
tasi & Clements, 1993; Roussos, Johnson, Moher, Leigh, Vasilakis, & Barnes, 
1999). Play, another factor prominent in games but less visible in CAI, has 
been supported in educational research as a necessary component of motivation 
(Lardinois, 1989, in Siemer & Angelides, 1995; Rieber, 1996; Romme, 2003). 
When people play, they allow for more efficient learning and cognitive mate- 
rial intake than they would in a more traditional setting (Baltra, 1990; Pange, 
2003). Thus, play has the potential to engage students in the learning process, 
which may lead to increased learning. Further, the level of interactivity between 
the user and program defines the depth of involvement of the user in the activ- 
ity (Bangert-Drowns & Pyke, 2001). Games are expected to have a higher level 
of interactivity compared to traditional CAI methods, again potentially leading 
to improved results. The amount of appropriate challenge in the program can 
also help scaffold the learning of the individual. If the challenge is too high, 
the learner may feel hopeless and quit trying, but if the challenge is too easy, 
then the learner may become distracted and lose interest. An optimal (and pos- 
sibly dynamic) level of challenge needs to be identified in order to capture the 
learner’s attention and interest and motivate them to continue to try to move to 
the next level of learning. Together, these three components (play, interactivity, 
and challenge) define a certain level of motivation. Rewards and scores are the 
final components of a game. They give feedback to the user in an attempt to 
both demonstrate correct versus incorrect answers, but can also act as a motivat- 
ing entity. When a student is oriented toward receiving more points or rewards, 
she/he may work faster and/or more efficiently. In an educational game, that 
would translate to more questions being answered or result in faster or more ef- 
ficient learning. 

Simulation in Gaming Technology 

Researchers have documented the effectiveness of simulation as a learning 
tool over the last 30 years (Farrel et ah, 2003; Lintern, Sheppard, Parker, & 
Yates, 1989; Lintern, Roscoe, Koonce, & Segal, 1990; Tkaz, 1998). Moreover, 
simulation has been identified as being particularly effective at training skills 
and procedures, as it allows learners to practice these behaviors in an artificial 
environment that mimics the real world. The utility of simulations in education 
has been examined at greater length elsewhere (Chaffin, Maxwell, & Thomp- 
son, 1982; Thurman, 1993), but extensive evidence exists showing that simula- 
tion is a highly effective way to communicate skills and operational knowledge 
to learners (Salas & Cannon-Bowers, 2001). Given that the use of simulation 
is a highly effective method of educating, researchers have argued that learning 
games incorporating digital simulation are ideally suited to learning in multiple 
domains, and may represent the future of learning in technological societies 
(Aldrich, 2004). The boundary between simulations and games can become 
blurred on occasion, but taxonomies exist to aid in understanding and differen- 
tiation between the two (Schmucker, 1999). 

Learning games may represent an effective way of delivering simulation to a 
wider population of learners than currently have access to such systems. A num- 
ber of studies have shown significant transfer of learning from digital games 
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to real world applications in a number of domains (Jentsch & Bowers, 1998; 
Jones, Kennedy & Bittner, 1981; Tkacz, 1998; Van Eck, 2000). However, sim- 
ulation-based games are associated with a number of drawbacks, including the 
difficulty and expense of developing such systems. Moreover, the open-ended 
nature and nonlinear presentation of material characteristic of simulations may 
not mesh well with the rigid structured environment of current classrooms and 
curricula. 

Teach-Test Methodologies 

Another avenue through which games increase effectiveness is in the design 
of the game itself and the way in which the game presents learning material. 
Systems such as CAI use non-interactive “teach-test” methodologies, trading 
the known advantages resulting from the use of simulation in learning for the 
decreased cost and development time and increased teacher acceptance of tra- 
ditional linear teaching methods. It is unknown whether this tradeoff in lower 
transfer-of-training results, based on teach-test methods for reduced costs and 
development time, will have the same results in games. 

Developers have created games for the educational market. So-called 
“edutainment” games have existed for decades (Prensky, 2001). Such games typ- 
ically use a very different structure from simulations, commonly presenting the 
material to be learned in a linear, predetermined manner, which is mediated by 
the overall structure of the game. This can be accomplished in many ways, but 
one common presentation format is to alternate segments of game-play with 
segments of learning (Prensky, 2001). 

Due to the high level of control over the structure and order of presentation 
of the material to be learned, such systems are much more easily integrated into 
classrooms and traditional learning environments. Additionally, it is probable 
that educators and administrators would more easily accept such systems, as 
they can more easily be made to support state- and federally-mandated educa- 
tion requirements. Equally importantly, these games are quicker, cheaper, easier 
to make, and they require less specialized expertise than simulation-based ap- 
proaches (Aldrich, 2004). Therefore, if it could be shown that such approaches 
to learning-game design were effective at engaging and enabling learners, these 
systems could be quickly and easily deployed in classroom and other learning 
environments around the country. 

However, it has not yet been shown that game-enabled instruction is effective 
using games that use a more traditional rather than simulation-based approach 
to learning. The aim of the present study is to examine whether a maximal ef- 
fort application of this type of approach, using virtual reality and incorporating 
the work of professional educators and system designers, could produce a learn- 
ing game that can increase learning in both hearing and deaf children beyond 
the improvement seen in simple CAI. 

Virtual Reality Technology 

The system used in this study utilizes virtual reality technology. VR for educa- 
tional purposes involves using three-dimensional figures on a computer to rep- 
resent accurate objects and scenarios. Navigation in any direction is possible, 
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and activities, games, or tests can be combined with scenes to create meaningful 
educational experiences. Two main benefits of VR over traditional two-dimen- 
sional CAI are the three-dimensional, life-like experience and the individuals 
control over the program. These make VR a distinct learning experience that 
can aid in comprehension of complex ideas and skills (Roussou, 2004). 

Special populations with different learning needs, such as deaf children, may 
particularly benefit from VR technology. It has been suggested that children 
who are deaf think in more concrete terms than their peers (Cromby, Standen, 
& Brown, 1996; Myklebust, 1964 in Passig & Eden, 2000a), making skills such 
as reading comprehension or higher-level mathematics very difficult to learn. 
Using VR with these non-traditional students exposes them to problem-solving 
and inductive thinking in less abstract ways, possibly allowing them to connect 
the abstract concepts with the concrete scenarios on the screen that represent 
those concepts. Further, research has demonstrated that using three-dimen- 
sional VR, compared to the conventional two-dimensional computer games and 
graphics, can significantly improve deaf children’s flexible thinking skills, or the 
ability to see problems from different points of view (Passig & Eden, 2000b). 
Without causing frustration in other people, it also provides redundancy of in- 
formation and control over the task, which may increase the child’s motivation 
for learning (Cromby, Standen, & Brown, 1996). 

The VR program used in this study has been tested in its original form as a 
CAI program. It was designed to help the children to link the idea of the ques- 
tions to the meaning of the question by demonstrating it in the VR program. 
For example, instead of simply reading a word problem that asked “If Bessie 
the cow has 2 quarts of milk, Callie has 6 quarts of milk, and Nellie has 3 
quarts of milk, who has the most milk?” the student sees the question, as well 
as several cows with milk, and bottles holding accurate amounts of milk. The 
pictures allow children to see the relationship between the bottles of milk and 
might help them to understand the concept of “most.” The results showed that 
using the program in this format lead to significant increases in understand- 
ing complex mathematical concepts such as geometry, spatial sense, graphing, 
and some areas of language arts such as reading comprehension (Vogel, Asberg, 
Vogel, & Bowers, 2006). However, transferring the knowledge to a wider range 
of testing in math and language arts showed fewer clear benefits. While scores 
in math were significantly improved, the same pattern was not present in the 
language arts section. This discrepancy has lead to further investigation by the 
researchers. 

The goals of the present study are two fold: first, to identify whether the use 
of a virtual reality learning game based on traditional linear teaching method- 
ologies can increase learning beyond the level of improvement expected from 
the use of a CAI format, and second, to determine whether hearing and deaf 
children benefit equally from the use of such technologies. 

METHODS 

Forty-four children ages 7 to 12 years from a public elementary school in 
Florida were included in this study. There were 25 females and 19 males. The 
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elementary school used for this study was chosen because it houses a unique 
computer lab devoted to testing VR programs for educational settings. Primar- 
ily, these programs are used with the deaf education program; however, regular 
education classes are utilized occasionally as well. Students were chosen in two 
ways. All the deaf students whose parents allowed them to participate were in- 
cluded. The hearing sample was chosen through a volunteer process. The teach- 
ers volunteered their students and then those students whose parents allowed 
them to participate were included. Teachers received no instruction on the com- 
puter; however, students were given basic directions. 

The original number of children in each grade included: 12 second graders, 

13 third graders, 9 fourth graders, and 10 fifth graders. The final analysis ex- 
cluded two children (one in third and one in fifth, both were deaf, one female, 
and one male) because of excessive absences. Eleven of the remaining children 
were deaf and 3 1 of the children hear normally. All teachers of the children who 
are deaf used the total communication method in their classrooms (a combina- 
tion of signing and speaking), and all the children who are deaf were taught in 
a segregated classroom environment. General Dynamics-Advanced Information 
System created the VR programs — one system as a CAI program and another 
system with gaming attributes. 

Prior to the beginning of the computer sessions, all children completed a 
pretest that covered language arts and mathematics. The subject areas tested 
included determining the main idea of stories, identifying relevant supporting 
details and facts, arranging events in chronological order, and algebraic thinking 
such as describing and analyzing patterns, relationships, graphs, symbols, and 
functions. Each test was 15 questions in length, and the children were given 15 
minutes per section to complete the tests. 

This study used a quasi-experimental unequal control group design to com- 
pare the two conditions. Students were split randomly and equally into either 
the control or experimental group. The control group used the CAI program 
while the experimental group used the program with gaming attributes. During 
the following two-week session, the children were involved with their respective 
computer programs for 10 minutes per day. Students were given basic instruc- 
tions about how the program works and then encouraged to navigate through 
the program individually. Acquisition and transfer of knowledge were tested us- 
ing a posttest similar in design to the pretest, but with different questions. Im- 
mediately following the last intervention on the computer, students were given 
the posttest in paper form. Both pre and posttests were delivered in this format. 
The tests were based on the standardized tests used in the state of Florida called 
the FCAT (Florida Comprehensive Assessment Test). Though no validity or 
reliability tests were conducted on the tests used in this study, their content, for- 
mat, and order mimicked that of the FCAT, which is highly tested each year. 

RESULTS 

Overall 

Using a one-tailed, exact probability test, participants across groups did not 
perform significantly differently on the posttest compared to the pretest in the 
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language arts section. However, in the math section, there was a significant 
increase in scores (p=0.032), suggesting that in either condition, students im- 
proved their performance on the posttest. Neither the control group nor the 
experimental group performed significantly differently in the language arts tests. 
Further, this study found no significant difference in change scores between the 
two groups. However, the control group performed significantly better on the 
math posttest compared to the pretest (p=.006), while the experimental group 
showed no significant difference. Thus, using a two-tailed, exact probability 
test, which compared the control group versus the experimental group in the 
math section, there was a significant difference in change scores (Mcont=1.15, 
Mexp=0.00), p=.044, suggesting that the children using the traditional CAI 
program learned more in the math section compared to those using the pro- 
gram augmented with gaming attributes (see Table 1). Each grade was com- 
pared separately. Table 2 contains a list of the results. 


Table 1: Overall Control vs. Experimental Group Difference Scores 


Group 

Difference Scores 

Test 

p-values 

Overall Language 

-.49 

One-tailed 

.090 

Overall Math 

.49 

One-tailed 

.032 

Control Language 

-.75 

One-tailed 

.086 

Control Math 

1.15 

One-tailed 

.006 

Experimental Language 

-.36 

One-tailed 

.278 

Experimental Math 

0.00 

One-tailed 

.500 

Delta (Exp vs. Control) Language 

— 

Two-tailed 

.637 

Delta (Exp vs. Control) Math 

— 

Two-tailed 

.044 


'Indicates significant restdt. 


Table 2: Individual and Comparison Difference Scores per Grade Level 


Grade 

Second 

Third 

Fourth 

Fifth 

Control Language 
(Difference Scores) (One-tailed) 

-0.67 

-1.17 

0.25 

-.125 

Control Math (Difference) 

1.00* 

2.17* 

1.25 

0.25 

Experimental Language 
(Difference Scores) (One-tailed) 

0.17 

-2.00 

0.60 

0.00 

Experimental Math 
(Difference Scores) (One-tailed) 

0.00 

0.50 

-0.20 

-0.40 

Delta (Experimental vs. Control) 
Language (p-values) (Two-tailed) 

.634 

.642 

.808 

.465 

Delta (Experimental vs. Control) 
Math (p-values) (Two-tailed) 

.249 

.181 

.356 

.893 


'Indie sites significant result . 
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Hearing Status 

The groups were divided by hearing status and compared. The difference 
scores in the deaf/hearing-impaired group from the pretest to posttest were not 
statistically significant in either the math or the language arts sections. In addi- 
tion, the difference scores in the hearing group for the math section were not 
statistically significant. However, in the language arts section, a decrease was 
found (collapsing across conditions) from pretest to posttest, suggesting that 
using the computer to practice reading skills, regardless of the condition, result- 
ed in reduced comprehension for children who hear normally but not for deaf 
children. Data indicated no significant differences between the difference scores 
of these two groups in the language arts or the math sections (see Table 3). 


Table 3: Difference Scores Compared by Hearing Status 


Group 

Difference Scores 

Test 

p- values 

Hearing Language 

-1.00 

One-tailed 

NS 

Hearing Math 

.26 

One-tailed 

.155 

Deaf Language 

.73 

One-tailed 

.166 

Deaf Math 

1.36 

One-tailed 

.064 

Delta (Hearing vs. Deaf) Language 

— 

Two-tailed 

.058 

Delta (Hearing vs. Deaf) Math 

— 

Two-tailed 

.091 


'Indicates significant result. 


Students Who Are Deaf 

Finally, this study made comparisons between the deaf control and deaf 
experimental groups. The deaf control group’s difference scores from pretest 
to posttest did not change significantly. However, there was a significant im- 
provement in the math section {p=. 023), suggesting that using the VR program 
with the CAI method resulted in improved math skills. No significant changes 
were observed in the experimental group in either the math or language arts 
sections. Further, this study found no significant difference when comparing 
the change scores between groups in the language arts section, suggesting that 
regardless of the type of program used, the performance of the deaf children 
did not benefit or decline because of using the VR program. However, a signifi- 
cant difference in change scores between groups was found in the math section 
(p=. 033), favoring the CAI version of the VR program over the gaming version 
(see Table 4). 

DISCUSSION 

The present study expanded existing methods of CAI to determine if they 
could be improved upon by presenting the same learning material in the same 
format, but embedding it in the structure of a video game. Wftile evidence 
supporting the effectiveness of learning games has previously been established 
(Jentsch & Bowers, 1998; Tkacz, 1998), previous studies have used games built 
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around simulation-related approaches. The goal of the present study was to ex- 
tend these findings to games that do not rely on simulation-based learning para- 
digms, but instead incorporate traditional educational approaches to structuring 
and presenting material. 


Table 4: Difference Scores Compared by Hearing Status 


Group 

Difference 

Scores 

Test 

p-values 

Deaf Control Language 

.20 

One-tailed 

.440 

Deaf Control Math 

3.20 

One-tailed 

.023 

Deaf Experimental Language 

1.17 

One-tailed 

.119 

Deaf Experimental Math 

-0.17 

One-tailed 

.421 

Delta (Control vs. Experimental) Language 

— 

Two-tailed 

.529 

Delta (Control vs. Experimental) Math 

— 

Two-tailed 

.033 


'Indicates significant result. 


Reading Skills 

The lack of any observed improvement in language arts skills regardless of 
condition or hearing status suggests that CAI may represent a less effective 
means of teaching language arts skills to children. This might be due to the way 
in which the material was presented in the pre and post test, or it might be a 
result of inadequate organization or poor presentation of the learning material. 
An alternative explanation is that language arts skills may be simply harder to 
teach, regardless of method, and that the participants did not use the system of- 
ten or long enough to obtain results. Additionally, the concrete representations 
of language arts may be more difficult to produce compared to mathematics. 
This may impede the impact of the visualization component used in VR to aid 
comprehension. 

One disturbing finding was that the performance of hearing children inex- 
plicably became worse in the post-test of this study. The effect itself was di- 
rectly attributable to the enormous drop in performance of the 13 third grade 
students in the study who had normal hearing. This finding would appear to 
suggest that the drop was due to the influence of a variable outside of the con- 
trol of the study. Third grade students in the experimental condition showed a 
large negative change score, while all other grades showed either no change or 
a modest increase in performance on the same test. It is noteworthy that in the 
state of Florida, all third grade students are required to read on grade-level to be 
promoted to fourth grade. This requirement generally leads to intensive reading 
work for this specific grade. As the current study was conducted at the end of 
the school year, it is possible these children were simply overworked in reading 
during their required classroom time leading to a decrease in concentration dur- 
ing the posttest phase. 
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Mathematical Skills 

The observation of a significant main effect with regard to math skills, such 
that participants’ math scores increased significantly from pretest to posttest 
when collapsing across condition and hearing status, would seem to suggest that 
CAI may generally be an effective method in training math skills, regardless of 
the approach used when designing the system. However, examining the effect 
further reveals that this overall difference is mostly due to a large significant im- 
provement in difference scores on math test performance from pretest to post- 
test in the control condition. The experimental group showed no such improve- 
ment, suggesting that the use of the learning-game format not only failed to 
improve math skills, but actually negated the improvement resulting from the 
use of the CAI system altogether. While participants in the experimental group 
did no worse than those in the control group, they did not improve significantly 
from pre to posttest. Participants in the control group did improve from pre to 
posttest. This finding is reinforced by the significant difference in change scores 
on the math test between conditions, such that there was a significant differ- 
ence between pre- and post-study math scores in the control group but not in 
the experimental group. Potentially, the use of the computer’s representation of 
mathematical problems allowed students, regardless of hearing status, to bet- 
ter visualize the problems. This concept has been supported in other settings 
(Chou, Hsu, & Yao, 1997; Drake, 2003; Gadanidis, Sedig, & Liang, 2004; 
Lowe, 2004; Ploetzner & Lowe, 2004), suggesting that the ability to concretely 
represent abstract concepts aids in comprehension of mathematics. 

Non-Simulation Games 

The current study failed to extend the findings of Jones, Kennedy, & Bittner 
(1981), Povenmire & Roscoe (1973), Lintern et al. (1989), Lintern et al. 

(1990), Tkaz (1998), or Farrel et al. (2003) to games utilizing non-simulation 
based content. The implications of this are significant to the future develop- 
ment of learning games. One interpretation of these findings is that games 
designed to enable learning should rely on a simulation-based approach rather 
than the linear and structured presentations traditional in educational environ- 
ments. This is unfortunate, in that simulation-based learning games are more 
difficult to incorporate into existing educational curricula than games relying on 
traditional educational approaches such as the one examined in this study. 

While more research into the utility of learning games as teaching tools is 
still necessary, this study suggests that attempts to integrate traditional linear 
presentation of material in learning-game format may be a less effective strategy 
than simulation-based approaches. While it is difficult to determine exactly 
why this is, a number of hypotheses can be proposed for further research. It is 
possible that the children enjoyed the game part but were bored by the learning 
material. Because of the sharp division between the learning and game content 
inherent in this type of design, the participants might have rushed through 
the learning material in order to reach the game more quickly. In doing so, 
they may have failed to devote adequate attention to learning. By contrast, the 
participants in the control condition did not have the experience of the game 
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condition to compare to the traditional CAI. Consequently, they might have 
found the novelty of the VR setup sufficiently engaging and motivating in and 
of itself, and therefore were more likely to pay attention to the learning mate- 
rial. One lesson that designers of learning game systems might draw from this is 
that learning material must be presented in a manner that is organic and non- 
disruptive to the game. This is much easier with simulation games in which the 
game content itself is educational, but it may also be possible to integrate learn- 
ing material into other types of games successfully. In order to work, however, 
such systems must ensure that the entire experience is engaging and motivating 
to avoid the effect observed in this study. 

Deaf Sample 

The findings specific to the deaf children who participated in the study largely 
mirrored the experience of the hearing participants. Deaf children did not ex- 
perience any improvement in language arts skills in either condition. They did, 
however, exhibit a significant improvement in math skills following the use of 
the system. Deaf children in the control group (who received non game-based 
CAI) improved more than their counterparts in the experimental group (those 
who used the game). Again, as the pattern is the same as observed in the hearing 
group, it is believed that the same issues have contributed to greater improve- 
ment in the traditional method versus the lower results in the game method. 

Impact of Learning Games 

It is important to note that these findings should not be interpreted as dis- 
counting the usefulness of learning games. Ample evidence exists supporting 
the usefulness of such systems in multiple learning environments, a small por- 
tion of which is identified in the introduction to this article. The present work 
does, however, seek to differentiate between learning games which are based on 
simulation technologies or other integrated learning tools and learning games 
which are based on traditional presentation of linear material interspersed with 
segments of game play. In essence, the connection between the material to be 
learned and the game play experience must be seamless. Otherwise, the system 
may not only fail to exploit the advantages of learning games, but may actually 
inhibit the ability and motivation of users to learn the material. This study has 
also identified few significant variations between a hearing population and deaf 
children with regards to the effectiveness of CAI and game-based learning. It 
can therefore be concluded that, insofar as the present study explored the in- 
teraction between learners and learning games, deaf children and their hearing 
counterparts have responded similarly to these types of CAI. Learning games 
represent a potentially useful technology for the classroom, but the design of 
such games must be carefully tailored to provide an experience that is engaging 
and motivating in order to avoid the effects observed in the present study. 
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